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ABSTRACT 



The Comprehensive School Reform Demonstration Program (CSRD) 
is designed to assist low-performing schools, especially Title I schools. 
Grants are awarded through statewide competitions. This report offers the 
findings of Mid-Continent Research for Education and Learning's analysis of 
the application process and characteristics of first-round applicants in the 
Central Region states. The characteristics of the schools granted funding 
were analyzed to determine whether CSRD funding is reaching the populations 
intended by the program. The data specifying school characteristics were 
obtained from the National Center for Education Statistics’ Common Core of 
Data. The types of technical assistance provided to potential applicants, the 
priorities in scoring applications, the major differences between successful 
and unsuccessful proposals, and the weaknesses of unsuccessful proposals were 
assessed. The success rate of low-achieving schools in obtaining funding and 
how this rate compares to other schools were also addressed. Rural and urban 
schools were compared by examining the percentage of schools that applied for 
funds, as well as the percentage that received funds. Analysis indicated 
substantial differences among states in the application process, specifically 
in the number of reviewers participating, the scoring rubrics applied, and 
the average funds awarded. Successful applicants had slightly fewer students 
and teachers, served a greater percentage of students eligible for free or 
reduced lunch, and more often demonstrated student need through involvement 
in Title I and school -improvement efforts. (RKJ) 
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INTRODUCTION 



In FY 1998, Congress appropriated $145 million for the Comprehensive School Reform 
Demonstration (CSRD) Program. This three-year program is designed to assist low-performing 
schools, especially Title I schools, in their school reform efforts. Individual schools receive a 
minimum of $50,000 per year, renewable for up to three years, for training, technical assistance, 
instructional materials, and parent and community outreach activities. Funds are made available to 
states and grants are awarded to Local Education Agencies (LEAs) through statewide competitions. 
Schools are required to adopt an externally developed model or develop their own internal model 
of comprehensive school reform. 

This study examined several policy questions about the characteristics of the LEAs which 
are submitting applications and the distinguishing characteristics of the proposals which are funded 
to determine whether CSRD funding is reaching the schools and populations of students intended 
by the program. The main research questions are as follows: 

► What types of technical assistance did states provide potential applicants? 

► What were state priorities for CSRD programs as reflected in their scoring 
rubrics for applications? 

► What are the major differences between successful and unsuccessful CSRD 
program proposals? What are the major weaknesses of the latter? 

► What is the success rate of schools specifically targeted for the CSRD 
program (e.g., “low-achieving schools”) in obtaining funding and how does 
this rate compare to other schools? What percentages of urban and rural 
schools are applying for and receiving CSRD funding? 

The purpose of this report is to describe the findings from McREL’s analysis of the CSRD 
application process and the characteristics of first round applicants in the Central Region states. 
McREL also collaborated with other regional laboratories on this policy research effort. Where 
appropriate, comparisons are made between the findings from the seven states in McREL’s service 
region and from the 28 states represented in the cross-laboratory effort. 1 



1 ButtramJ.L, Castenada-English, P., Saenz, V., Hauser, B., Moats, S., Lane, B., Keller, B., Friedman, L., 
Zuckerman, D., Chang, J., Church, M., Davis, B., and Huebner, T. (2000, October). Analysis of National 
Comprehensive School Reform Demonstration Program Competition. Austin, TX: Southwest Educational 
Development Laboratory. 
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METHOD 



This study involved analyzing information from both successful and unsuccessful CSRD 
applications collected from McREL’s seven-state service region. Application guidelines, proposals, 
and scoring data were requested from the CSRD program coordinators in the Central Region states: 
Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota, and Wyoming. Information 
regarding the characteristics of the applicant schools and the results of the application process was 
compiled in a database along with additional school information obtained from the National Center 
for Educational Statistics’ Common Core of Data (CCD). Descriptive analyses focused on 
comparisons between the characteristics of the schools and proposals that were successful with those 
that were not. 2 



FINDINGS 

Several aspects of the CSRD competition were examined and are summarized: the technical 
assistance offered by states provides a context for the application process; the applications received 
and awards made in round one comprise the data set studied; the scoring systems used by states 
establish relative priorities and determine success; and the school characteristics of successful and 
unsuccessful applicants provides evidence of whether target schools were reached. 

Technical Assistance to LEAs . 

Various types of technical assistance were offered by state agencies to potential applicants 
during the CSRD competitions. McREL often collaborated with states in planning and/or delivering 
these services. Table 1 shows the types of assistance offered and the number of Central Region states 
that offered each type. As shown in the table, states offered multiple technical assistance options for 
applicants. All seven central states offered self-help resources and consultation via telephone and 
e-mail; all seven also held state- or regional-level planning workshops for potential applicants. Two 
states offered on-site technical assistance to potential applicants. 

Table 1 also includes data from the 28 states that participated in the cross-laboratory study. 
While the availability of technical assistance was relatively higher among the Central Region states 
than among the larger cross-laboratory sample, the most frequent types of assistance offered were 
the same. It is important to note that these samples of states are not independent, however, since data 
from the states in McREL’s region were also included in the larger cross-laboratory data set. 



2 A second phase of this policy study was initially designed to identify and contact schools that were 
“targets” of the CSRD Program but did not apply for funding. However, the cross-laboratory team determined that 
such an effort would not yield useful information because of the amount of time that elapsed between the first round 
of CSRD funding and the proposed data collection. 
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TABLE 1 

States Offering Technical Assistance to LEAs 



STATES THAT OFFERED ASSISTANCE 



TYPE OF TECHNICAL ASSISTANCE 



Central Region Cross-Laboratory 

(N=7) States* (N=28) 



Provided resources for schools (e.g., web sites, guides) 


7 


100% 


20 


71% 


Telephone consultation (question/answer, referrals) 


7 


100 


20 


71 


E-mail consultation (question/answer, referrals) 


7 


100 


18 


64 


State level planning workshop 


6 


86 


16 


57 


Regional level planning workshops 


5 


71 


12 


43 


Showcase (model developer fair) 


4 


57 


13 


46 


Site visits 


2 


29 


8 


29 


District level planning workshops 


0 


0 


3 


11 



Buttram et al. (2000) 



Application Review Process 

Proposal Scoring . The procedures, criteria, and numbers of reviewers used to score 
proposals varied by state. One state used five reviewers per proposal, most used three reviewers, and 
another used only two. Three states used essentially the same scoring rubrics. Comparisons across 
these disparate scoring systems were difficult. However, it was possible to compare the scoring 
rubrics and the relative emphasis that state systems placed on them. 



Figure 1 shows the distribution of 
mean possible scores as percentages of the 
overall total. As shown in the figure, across 
states 67 percent of the available points 
were allocated to the nine CSRD criteria, 
19 percent to administrative planning and 
budget, and 14 percent to priority points. 

Priority points. All seven states 
offered priority or bonus points to 
applicants who provided evidence of high 
need or low-performing status. The 
maximum percent of priority points ranged 
from a low of 6 percent to as much as 22 
percent of the overall proposal score. 



Figure 1 . Overall Scoring Distribution 
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Typically, schoolwide Title I schools were eligible for more points than were targeted Title I schools 
or non-Title I schools. 

Although the number of applications in the Central Region that received funding as a result 
of receiving priority points is not known, the cross-regional study suggested that these points made 
little difference in awards (Buttram et al„ 2000). One possible explanation is that the prospect of 
priority points may have encouraged schools to apply that were needy but otherwise not well- 
prepared to respond to the funding opportunity. 

CSRD criteria. The largest proportion of the overall scoring was allocated to the nine CSRD 
criteria identified in the enabling legislation. Figure 2 shows the average relative distribution of 
points to these specific categories for the states in McREL’s region. Note that because (a) the 
categories are not independent of one another and (b) the scoring rubrics used by states did not 
always directly align with specific categories, these data are approximations. As shown in the figure, 
the criteria given the greatest weight overall 
were “Employ innovative strategies and 
proven methods for student learning, 
teaching and school management that are 
based on reliable research and effective 
practices and have been replicated 
successfully in schools with diverse 
characteristics” (19%) and “Have a 
comprehensive design for effective school 
functioning including instruction, 
assessment, classroom management, 
professional development, parent 
involvement and school management” 

(15%). Those given the least weight were 
“Ensure support of school faculty, 
administrators and staff’ (6%) and “Use 
high-quality external technical support and 
assistance from a comprehensive reform 
entity with experience or expertise in schoolwide reform and improvement” (7%). 

Administrative. Administrative points were given for a variety of other reasons. These 
included a reasonable, cost effective budget; plans for project implementation; evidence of 
commitment by either the model developer or other key staff; project abstract; dissemination plans; 
and administrative details such as including a cover page and assurances. 

Applicant Scores . As previously noted, while states provided detailed information about 
their scoring systems and procedures, the information available to McREL regarding the results of 
their reviews was limited. Several Central Region states reported complete item and re viewer ratings, 
while one reported only total scores and another did not report scores. Scores for all unsuccessful 



Figure 2. CSRD Criteria Scores 
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applications were unavailable for two states, and for two applications in another state. Because of 
the limited nature of the data, analyses of applicant scores were not conducted. The cross- laboratory 
effort was able to analyze scoring data from 22 of the 28 states in its sample and found that the 
profiles for successful and unsuccessful applicants were very similar (Buttram et al., 2000). 

A pplications and Awards 

Table 2 summarizes the first round CSRD applications and awards made in the seven Central 
Region states. As shown in the table, a total of 165 proposals were submitted on behalf of 196 
schools and 56, or about one-third of them, were successful. In one state the application period was 
extended so that all applicants could have additional time to review and improve their initial 
proposals; in another, several rounds of competition were implemented. While those that were 
unsuccessful in the first round were allowed to resubmit and receive funding during subsequent 
rounds, only the first-round results were included in this study. The success rate for the 28-state 
cross-laboratory sample was somewhat higher, about 50 percent (Buttram et al., 2000). One likely 
reason why the overall success rate in the cross-laboratory sample was higher than that for the 
Central Region is that the cross-laboratory sample included data from subsequent rounds of funding. 

TABLE 2 

Central Region CSRD First Round Applications 



N 

Row Percent 


SUCCESSFUL 


UNSUCCESSFUL 


TOTAL 




56 


109 


165 


APPLICATIONS 


34% 


66% 


100% 




64 


132 


196 


SCHOOLS 


33% 


67% 


100% 



Amount of Funding . The round-one awards in the sample totaled $3,920,663, or just over 
half of the funding available to the states in the first year of the Program. This amount translates to 
approximately $62,200 per participating school, although the average size of the awards varied 
substantially by state, ranging from about $102,000 per school to $28,700. The relatively small 
awards made to some schools were apparently the result of proposals submitted jointly by two or 
more schools. The amount of funding requested per school was comparable between successful and 
unsuccessful applicants. 

School Reform Models . A wide variety of models were proposed by the applicants. Multiple 
models were frequently identified which allowed schools to select complimentary approaches and 
components that best meet their needs. Those specific CSRD models mentioned most often included: 
Success for All, Roots and Wings (often in conjunction with Success for All), Accelerated Schools, 
Coalition of Essential Schools, Expeditionary Learning, and Multiple Intelligences. Many schools 
also proposed “homegrown” models of their own design. However, these homegrown models were 

5 
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generally less successful in securing awards. A separate report describing the population of 
alternative models in the region and factors that might guide services to these schools was recently 
prepared by McREL. 3 

School Characteristics 



Selected characteristics of the central state applicant schools obtained from the NCES 
Common Core of Data are summarized in Table 3. For these analyses only data from those six states 
that provided information for both successful and unsuccessful proposals were included. As shown 
in the table, nearly three-fourths (74%) of the applicant schools serve students at the elementary 
level. The remaining schools are about equally divided between the middle (11%) and high school 
levels (9%); a smaller percent (6%) serve various other combinations of grades. Approximately half 
(47%) of the applicants are located in rural communities or small towns and a third (34%) are in 
urban settings. There were no significant differences in the distributions of successful and 
unsuccessful applicants on the basis of either grade levels or locale. 

School Size . Overall, applicant schools averaged 419.6 students and 26.2 FTE teachers, or 
about 16 students per teacher. Although the student/teacher ratios were comparable for successful 
and unsuccessful schools, applications from smaller schools appeared to be slightly more successful. 
Unsuccessful schools averaged 450.9 students and 28.2 teachers while successful schools averaged 
354.0 students and 21.9 teachers. Both of these differences were marginally significant. 4 

\ 

Student Ethnicity . As shown in Table 3, slightly more than one-third (36%) of the students 
from all applicant schools are members of minority groups. Black and hispanic students comprise 
the largest percentages of minority students (14% each), followed by American Indian/Alaskan 
natives (7%) and Asians (2%). While the percentage of minority students was somewhat higher 
among successful schools than unsuccessful schools (43% versus 33%), the difference was not 
statistically significant. 

Title I Eligibility . Nearly half (46.2%) of all students in the applicant schools were eligible 
for free or reduced lunch. This percentage was marginally higher among successful schools (5 1 .6%) 
than unsuccessful schools (43. 6%). 5 Table 3 also shows the percentages of Title I schools and 
schools identified for school improvement identified in the applications. Nearly all (97%) of the 
funded schools for which information was available are Title I and most of them were identified as 
in need of improvement. 



3 Clark, G., Apthorp, H., Van Buhler, R., Dean, C. & Barley, Z. (2000, August). Beyond the List: Schools 
Selecting Alternative CSR Models. Aurora, CO: Mid-continent Research for Education and Learning. 

4 t 187 = 1 .94, p = .05 and t 18 , = 2. 1 , p = .04, respectively. 





5 



t, 86 = 2.10,p = .04. 



Cross-laboratory Schools . The school characteristics of Central Region applicants were 
generally consistent with those from the 28 state cross-laboratory sample. In both groups, awards 
were made most often to schools at the elementary grade levels, schools that have Title I programs, 
and schools that serve relatively high numbers of students who qualify for free or reduced lunch 
programs. Given the predominately rural nature of the Central Region, it is not surprising that the 
schools in the cross-laboratory states were more often located in urban settings (Buttram et al., 
2000). No information on school size and student ethnicity was available for the cross-laboratory 
schools. 
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* CCD (1998) data availability varied; the actual number of schools represented for a particular characteristic may be less than the possible N for the 
sample. ** Significant difference between funded and non-funded applicant schools. 



SUMMARY 



A study of the first round of the CSRD competition in the seven Central Region states was 
conducted, based on the applications submitted, state guidelines, and information from the National 
Center for Educational Statistics. Comparisons with comparable data from the 28-state cross- 
laboratory study were also made and were generally consistent with those from the Central Region. 

Various types of technical assistance were offered by state agencies to potential applicants 
during the competitions. The findings show that the application process varied substantially among 
the states in terms of the number of reviewers who participated, the scoring rubrics applied, and the 
average funds awarded. While all applications requested at least $50,000 in support, the amount 
awarded per school was sometimes less than the minimum because of joint proposals. 

Analyses of the state scoring systems revealed that, across states, 67 percent of the available 
points were allocated to the nine CSRD criteria specified in the enabling legislation, 19 percent to 
administrative planning and budget considerations, and 14 percent to priority points based on need. 
Among CSRD criteria, the greatest weight was given to applications that employed innovative 
strategies and proven methods based on reliable research and effective practices and that presented 
a comprehensive design for effective school functioning. 

Nearly three-fourths of the 195 applicant schools were elementary schools and about half 
were located in rural communities or small towns. About one-third of the 165 first round applications 
in the Central Region were successful. The characteristics of applicant schools appeared to be 
consistent with the competition guidelines. As compared with unsuccessful schools, successful 
schools had slightly fewer students and teachers, served a greater percentage of students who are 
eligible for free or reduced lunch, and more often demonstrated educational need through their 
involvement in Title I and school improvement efforts. 
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